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An Evaluation of an Innovative Use of Computers 
and Volunteers in 7th Grade Math Instruction 



This paper summarizes a formative evaluation of a developmental project designed to 
maximize the potential use of computers in math instruction through the use of volunteers. 
The report focuses on concerns, issues and questions a practitioner should examine when 
replicating a similar project. The following paragraphs present a rationale and description 
of the project. 

One of the most effective uses of computers in the instructional process is during 
student practice. Computers can be used to ensure a very sophisticated drill and practice 
session, much more sophisticated than a teacher could implement. Computers are very 
efficient and effective at managing student practice. They can keep track of the number of 
times students have tried an item, how often they have reviewed an item, give students a 
small working pool of items consisting of new and review items, eliminate the sequential 
learning effect by randomly rearranging items, and many more tasks that a teacher cannot 
possibly manage or remember for 25 students or more (Merrill, 1986). In addition, 
computers can foster or maintain a high degree of learner attention and motivation because 
of the novelty and the gaming properties computers can build into a practice session. 

Another strength of computers in assisting in the instructional process is giving 
immediate feedback to the learner during practice. Teachers cannot give individual 
feedback to 25 or more students in a class consistently and frequently during student 
practice. Due to the interactive capabilities of the computer, however, students can receive 
immediate feedback on all their practice exercises. In fact, some software programs will 
give remedial help to students if their answers were not correct. 

Despite the strengths computers have in assisting instruction, some public schools 
have not maximized the use of the computers in their math instruction. Some schools use 
computers in their math instruction; however, they lack a management system which 
ensures that students are executing math programs commensurate to their particular math 
deficiencies. In some schools teachers send nart of their class to the computer lab to 
practice math exercises, however, students frequentiy end up playing math games which do 
not map to their math deficiencies or the practice they need. Usually the computer specialist 
supervising the computer lab doesn't know what specific programs each student should 
execute to meet his or her instructional needs. It is almost impossible to expect a teacher to 
manage some sludents practicing in the computer lab while oLher students are doing 
individual work in the classroom. Usually the teacher cannot send all the students at one 
time so he or she must remain in the classroom with ttie students who cannot go the 
computer lab. Even if a teacher has a small class and there are enough computers for each 
student in the computer lab, one teacher cannot effectively manage tiie lab. In short, some 
schools are not maximizing the computer because they do not have an effective 
management system which ensures that the students execute appropriate software for their 
particular learning needs. 

The main focus of this project was to use community volunteers to ensure that an 
appropriate management system was functioning so that students could practice appropriate 
math skills for their particular math deficiencies. In other words, through the use of an 
effective management scheme, the full potential and power of the computer would be 
maximized in helping students practice basic msth skills. 

This project was called the Productivity Math Project. The project was implemented at 
an Intermediate school in Utah. The primary stakeholder was a district Supervisor of 
Educational Support Services. 
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FUNCTION OF THE EVALUATION 



The object (evaluand) of this evaluation was the Productivity Math Project. The 
function of this evaluation was formative. In other words, the purpose of the evaluation 
was to provide information that would be used to improve the program. The data from this 
pilot class were not used for summative purposes, such as comparing the results from tliis 
pilot study to other programs to see which programs did better. The results from this study 
were used in working out the bugs in the project and developing the best possible program. 
Conclusions and recommendations focus on how the project can be improved so that the 
best possible program can be implemented in the future. 

EVALUATION QUESTIONS 

The evaluation addressed the following questions: 

1 . To what extent was the Math Productivity Project implemented according to the 
implementation design? 

a. How was the project designed to be implemented? 

b. What actually happened in the implementation of the project? 

2. ^Tiat impact did the project have on the students? 

a. To what extent did students achieve basic math computational skills (addition, 
subtraction, multiplication, and division)? 

b . To what extent did the project influence the students in terms of: 

(1 ) Their attitudes towards math? 

(2) Their anrJety towards math? 

c. To what degree did the project affect math homework study habits of the 
students? 

d . To what extent did the project provide students with positive reinforcement? 

e . To what extent did the project provide students with individual help from an 
adult on their math problems? 

3 . What were the strengths and weaknesses of the project as perceived by the 
participants and what suggestions did they have concerning how the project can 
be improved? (Participants included the volunteers, the teacher of the pilot class, 
the volunteer coordinator. The Intermediate School Vice-Principal, and students 
in the pilot group.) 

EVALUATION METHODS AND PROCEDURES 

Subjects 

This pilot study was conducted using a 7th grade class at an Intermediate School in 
Utah. The class was one of eight basic math classes being taught at the school for students 
who did not have the necessary basic matii skills to enroll in a regular seventh or eighth 
grade math class. As mentioned previously, these students were identified by one or more 
of tiie following criteria to be enrolled in a basic math skills class: 

a. Their results on the Metropolitan Achievement Te'^t. This test was given in the 
spring of 1986. Students who scored at least two grade levels below the normal 
grade level were placed in the basic math classes. 

b. Their results on the California Achievement Test. This test was also given in the 
Spring of 1986. Students who scored a least two gr'tde levels below the normal 
grade level were placed in the basic math classes. 

c. Their results on the WRAT Test. This test is given to incoming students not 
having a current test score. Students scoring low on this test were placed in the 
basic math classes. 
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d . As a result of teacher, counselor, and/or parent recommendations, students were 
placed in the basic math classes. 

Data were collected from 18 students in the class. However, data from six students 
were not used in the data analysis because these students either quit school early and we 
could not obtain posttest results, or students wei-e added to the class halfway through the 
study and we could not obtain pretests results. The data analysis was conducted using data 
from 12 of the 18 students. 

Instrumentation 

A number of various instruments were used to collect the appropriate data necessary to 
answer the evaluation questions. Each instrument used will be discussed in the following 
section. 

The evaluators developed interview schedules designed primarily to obtain qualitative 
data for answering the first and the last evaluation questions. These interview schedules 
were used by the evaluators in personal interviews with the volunteers, students, and 
school personnel. In addition, a few of the questions were designed to obtain perceptions 
from those interviewed about the achievement and learning gains of the students involved 
in this project. 

The evaluators developed the "Volunteer Tutoi' Activity Sheet" to obtain information 
concerning what type of daily activities or tasks the volunteers performed while they 
worked at the school. This instrument was designed to give information to determine to 
what extent the project was implemented according to the original design. The volunteers 
were requested to report their daily activities on one of these sheets before thej' left the 
school. 

The evaluators collected additional qualitative data necessary to answer the evaluation 
questions through conducting field notes. A limited number of observations were 
conducted at the site (the computer lab and the classroom) to help give the evaluators a 
better understanding of what was happening during the study. Only a few observations 
were scheduled due to budget constraints. 

Four criterion-referenced tests were used in the study, each containing multiple 
domains defined as units in the tests. These tests measured simple addition and subtraction 
skills (Beginning Mathematics 1), simple and complex multiplication and division skills 
(Beginning Mathematics 2), basic computation skills in solving fractions (Beginning 
Fractions), and basic decimal computation skills (Beginning IXicimals). These tests are 
referred to throughout the report as the Metra tests because they were published by Metra 
Publishing Company. 

An instrument developed by Richard Sudweeks called "My Feelings about School 
Subjects" was used to measure student attitudes about math. Dr. Sudweeks, Assistant 
Professor of Instructional Science at Brigham Young University, developed this instrument 
while he was working at Syracuse University in New York. The results of this instrument 
have been tested previously with students to ensure a high degree of validity and reliability. 
The instrument includes items measuring student attitude: about Math, English or 
Language Arts, Social Studies and Science. Scales measuring other subjects were included 
to ensure that if math attitude changed, it was independent of other school subjects, and not 
the result of a change in attitude about school in general. 

The Syracuse Math Anxiety Scale was used to measure student anxiety towards math. 
This insirumeni was developed by Sudweeks and Stroller at Syracuse University. The 
scale was originally designed f r upper high school students. In consultation with Dr. 
Sudweeks about using this instrument with seventh and eighth grade students, he felt the 
instrument would not need to be revised, even though it was designed for high school 
students. 

A questionnaire was developed by the evaluators to determine students' homework 
study habits, individual help they received during the intervention, and other important 
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information to answer the evaluation questions. The evaluators developed this instrument 
in consultation with the disirict Supervisor of Educational Suppon Services and the school 
administrators. 

*rhe R.AT. standardized test was used to measure pretest and posttest results in 
math achievement. 

Procedures 

The data collection phase of the evaluation was conducted from January to August, 
1987. The students began using the Math Computer Lab by the second week of January. 
In addition, the volunteers became involved with the students about the second week of 
January. 

To obtain the necessary ?.iformation to answer the evaluation questions, pretests and 
posttests were administerea to determine what affect the project had on the students. The 
Metra tests were administered by the volunteers. In an orientation meeting held in 
December 1986, or e of the evaluators trained the volunteers on how to administer the 
criterion-referenced tests to the students. This orientation meeting was held in December so 
the volunteers could administer the tests the first two w^ks of the second semester. 
Although much of the testing was done in the first two weeks of school, some testing was 
nOt completed until tlie first week in March. 

The Syracuse Math Anxiety Scale, h\c Attitude About School Scale, and the Student 
Questionnaire were administered to the students by one of the evaluators on January 23, 
1987. It took about 50 minutes to administer these three instruments. These instruments 
were administered about two weeks after the project began due to changes in the proposal 
and evaluation design. The students were asked to fill out the questionnaire assuming it 
was the first day of school and not the 23rd of January. That is, they were asked to fill out 
the questionnaire as if it had given to them on the first day of class of the second semester. 
This request of the students was necessary so the first two weeks of the second semester 
would not influence their responses concerning what previous experience they had had 
with a computer and individual feedback and reinforcement they had previously received 
from adults in their math class. 

We conducted a few observational sessions at the site. Only the volunteer coordinator 
was told ahead of time that we were coming to the Math Computer Lab to observe so that 
the project could be observed as it was naturally occurring. In addition to observational 
field notes, we informally interviewed some of the participants in the project, such as 
volunteers, the teacher, and some of the students. The observations and informal 
interviews were conducted to determine how the project was actually being implemented. 
In addition to the observations and interviews, we also attempted to determine to what 
extent the project was being implemented by asking the volunteers to record what type of 
things th?y did everyday at the school on the Volunteer Tutor Activity Sheet. Also, 
document analysis was also conducted during the projev'^t on various documents available. 

All of the posttests were administered on May 5, 1937. The Syracuse Math Anxiety 
Scale, Attitude about School insttument, and the Student Questionnaire were administered 
by one of the evaluators to the sttidents during the normal class period (50 minutes). These 
three instruments were exactly the same as the insttuments administered in January, no 
revisions had been made to them. The Metra posttests were administered throughout the 
entire day. Instead of having the volunteers give the posttests as they had done in the 
pretests, we decided to collect all the posttest data in one day. This way the data collection 
would not continue over multiple weeks as it had done in the pretesting. To accomplish the 
task of collecting all the Metra posttest data in one day, five evaluators conducted the testing 
until all the testing was completed. All of the evaluators were trained how to administered 
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the tests before testing the students. All four Metra tests v/ere given to each student It 
took about an hour to an hour and a half to test ea^^^ student. 

On May 21, 1987, we conducted interviews with the students and the teacher of the 
pilot group. We used the interview schedules to guide the interviews. Each of the student 
interviews took about 20 minutes and the teacher interview lasted about an hour. Other 
interviews with school personnel and all six volunteers were conducted over the phone. 

R.A.T, standardized tests scores of student achievement were collected from the 
teacher a few months after school was out for summer. These data were collecied because 
school personnel felt that these data showed significant learning gains between pretests and 
posttests results. 

FINDINGS 
How the Project was Implemented 

The Original Design of the Project 

The main focus of this project was to use community volunteers to ensure that an 
appropriate management system was functioning so that students could practice appropriate 
math skills for their particular math deficiencies. In other words, through the use of an 
effective management scheme, the full potential and power of the computer would be 
maximized h helping students practice basic math skills. T:e following items describe 
how this purpose or focus of the project was to be achieved in the original design of the 
project. 

• One of the first tasks is to determine the classroom management organizational plan. 
This plan involves determining the class instructional objectives, activities that need 
to be conducted to achieve the instructional objectives, identify who is responsible, 
and determine what materials are required. 

• Once the instructional objectives (for basic math computational sldHs) for the class 
are determined, appropriate computer software needs to be identif xi and purchased 
which maps (matches) directiy onto tiie instructional objectives. The software 
should be appropriate drill and practice applications, not just flashy, whistles and 
bells math games. 

'* Appropriate criterion-referenced tests which also map directly to the instructional 
objectives need to be purchased. These tests will be used to determine what math 
skills students are deficient. These tests will also be used in conducting pretests and 
posttests to determine achievement gains of the students in the pilot study. 

• ^Tie teacher will provide tiie volunteers with the information concerning what basic 
computational skills the students are weak in or lack mastery level skills. The 
teacher will obtain tiiis information from the results of the criterion-referenced tests. 
A file should be kept on each student in tiie class. This file should contain 
information concerning what math skills students needs to practice and reports on 
how tiiey are progressing, both from tiie teacher and tiie volunteers. 

• The student files will be placed in the Math Computer Lab so tiiat volunteers can 
deteraiine what math skills tiie students need to practice. The heart the project 
involves the volunteers determining (referring to the student files) what math skills 
each student needs to practice, and then helping the student execute the appropriate 
drill and practice software. When a student comes into uie Ivlaih Computer Lab, the 
volunteer should first refer to the student's file to determine what math skills that 
student needs to practice. After the volunteer has determined what skills the student 
should practice, the volunteer should then detennine (from a chart discussed in the 
following paragraph) what software program would be most appropriate for that 
student to execute. The Volunteer should help the student start the program and then 
allow the student to run the program with his partner. 
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• As mentioned previously, sofrA^are will need to be obtained which maps directly 
onto the instructional objectives and criterion-referenced tests. A chan will need to 
be developed that maps the basic math computational skills to the appropriate 
software to practice for that math skill This way, when the volunteer determines 
what skills a student needs to practice, he will refer to this chart and immediately 
determine which software program he will help the student execute to practice that 
particular program. 

• Volunteers will be available during the computer lab session to help students 
individually after they help each student execute the appropriate software program 
for their instructional needs. There should be at least two volunteers supervising the 
computer lab. 

^ Twelve students will go to the Math Computer Lab twice a week. Since there are 
more than 12 students in the class, students will have to rotate going to the lab. 
Because there are only six computers in the lab, two students will work with one 
computer. Students will receive direct instruction from tiie teacher twice a week in 
the classroom. One day a week will include testing or leview in the classroom. The 
teacher will update tiie student files on their practice needs according to the results of 
frequent testing. 

• The Matii Computer Lab will be made availaole at 2:10 pm on Tuesdays, 
Vx^ednesdays, and Thijrsdays during the 8tii hour. The eighth ho'ir is an optional 
class time for students to catch up on homework and other necessary things they 
need to do. Volunteers will supervise the lab during these hours for students to 
execute software programs and receive tutorial help, 

• The volunteers will manage tiie record keeping system in the computer lab. 
Appleworks and M.E.C.C. software management systems will be used to record 
and manage all tiie necessary student data. If the volunteers cannot perform this 
task, a resource aide will be hired to manage tiie record keeping system. This recoid 
keeping system will record such data as when students have achieved mastery of 
matfi concepts and skills, what software programs they have executed, what 
individual help the students have received, etc. 

What Actually Happened in the Implementation 

The project was not completely implemented as it was designed. In fact, the key 
components of the project never did occur. The key components were tiiat tiie volunteers 
would select the appropriate software programs which each student should execute to give 
him the necessary practice and feedback to meet his instructional needs. Each of the 
components of tiie project design explained in tiie previous section will now be addressed 
concerning what actually happened to tiiose items in the actual implementation of the 
project. These findings were determined from data collected from interviews, observation 
and document analysis. 

• School personnel did develop a classroom management organizational plan. This 
plan consisted of a one page sheet summarizing the activities that needed to be 
conducted to achieve tiie instructional objectives, identified who was responsible, 
and determine v>'hat materials were requireo ''or each of the major instructional 
objectives. In addition, die plan included attached sheets which listed the specific 
math concepts to be taught (for tiie scope and sequence of the class) and for each 
specific math concept, a section on the activities, who is responsible and materials 
needed. In addition, the plan also included some attached sheets showing tiie CM 
Software programs and curriculum match. These sheets listed software programs 
and then match the appropriate math concepts that these software programs 
addressed. These sheets identified most of the matii concepts tiiat were to be taught 
in the class; however, some math concepts were not identified. 
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• As mentioned previously, school personnel identified many software programs 
which matched the specific math concepts and instructional objectives for the class. 
However, tlieir CAI Materials and Curriculum Match chart was incomplete, because 
some of the math concepts did not match any of the software identified. Even 
though many software programs were identified, very few software programs were 
purchased by school personnel. At the beginning of the project, five disks were 
available in the Math Computer Lab. By the end of the project, 10 disks were 
available. 

• As was mentioned previously in the report, appropriate criterion-referenced tests 
which matched the math concepts and instructional objectives of the class were not 
purchased by the school personnel. Criterion-referenced tests published by Metra 
were decided to be used at the last minute. These tests did not completely match the 
math concepts of the class; nevertheless, because they did measure '^asic math 
computational skills, they were used in the study. 

• Information concerning individual student weaknesses and instruction J needs were 
not provided to the volunteers. The teacher did record students' instructional needs 
as a result of the criterion-tests administered to the students; however, the volunteers 
did not have access to these records in the Math Computer Lab. If a volunteer was 
available to tutor a student, the teacher would review with the volunteer what skills 
the student needed to practice and give the volunteer the student file to refer to while 
tutoring the student. The volunteer then returned the student file to the teacher 
following the tutoring session. 

• The main component of the project was not implemented. That is, the volunteers did 
not select appropriate software for each student to execute, depending on the 
student's individual instructional needs. However, the teacher did tr\' to execute 
appropriate software for each student at the beginning of tlie lab session. The 
teacher would normally take 12 students to the lab on Tuesdays and Thursdays. She 
would allow students to pair up with a friend using one computer. She would then 
give each student appropriate software for their particular instructional needs. After 
she started all the students on the software programs they should have executed for 
their instructional needs, she would go back to her classroom and work with the 
other students in the classroom. Some of the students told us during our interviews 
that many students would wait for the teacher to leave, take out the disk she had 
given them and insert their favorite game disk. Apparently the volunteers 
supervising the lab either did not care if the students switched disks or were unaware 
of what the students were doing, according to some of our informants. 

• The volunteers did not refer to a student file to determine what math skills a student 
needed to practice, nor did they refer to the match chart showing the match between 
math skills (or concepts) and appropriate software because these resources were not 
available to them. Midway through the project, the evaluators recommended to the 
school personnel that if they performed the following three tasks, it would have a 
significant influence on the project: 1) copy the teacher's student files she had 
documented listing the specific math deficiencies of each student and make these files 
available in the Math Computer Lab, 2) have someone who was computer literate 
review all the software programs in the lab and match each of the programs with the 
scope and sequence instructional objectives (and if he or she had time, to purchase 
more math drill and practice software which would match to the class objectives), 
and 3) train each of the volunteers basic computing skills so they could manage the 
lab on their own. These suggestions were not implemented. 

• Even though the volunteers were not able to help the students load and execute their 
programs, Uiey did help and tutor the students upon request in the lab. From 
observational and interview data, it appears that the volunteers spent most of the time 
in the lab watcning over the shoulders of the students and helping them when they 
asked for help. Some of the students said during the interviews that the volunteers 
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were very helpful when they couldn't understand how to do a math problem. At the 
beginning of the project, there were usually at least two volunteers supervising the 
lab on Tuesdays and Thursdays during the regular class period. However, alx)ut 
midway through the project had to drop out of the study, which left one volunteer 
supervising the computer lab by himself. More will be discussed concerning the 
activity of volunteers in the "Other Implementation Issues" section. 

• The students attended the lab twice a week (unless they had to rotate since there were 
more than 12 students in the class) and had direct instruction with the teacher in a 
traditional classroom situation three times a week, A few of the volunteers never did 
supervise students in the computer lab. These volunteers came to class during the 
regular class time and tutored students. The teacher would determine which student 
needed the most individual help that day and send that student with the volunteer to 
be tutored during the entire class period. 

• The 8th hour did not involve students practicing math skills on the computer the 
extent it was envisioned in the original design. Students usually attended the 8th 
hour because they were required to by their teacher. The volunteer who usually 
worked the 8th hour said that most of her time was spent administering the Metra 
tests, and once in a while tutoring a student. 

• Appleworks or another similar M.E.C.C. database program were not used in the 
project. The software was never purchased for the project and the volunteers 
wouldn*t have known how to use it even if it was available. 

Impact of Project on Students 

This section of the evaluation report will identify specific evaluation questions 
concerning what impact the project had on students, the methods used to collect data to 
address those questions, and a summary of the findings of the data analysis. In analyzing 
the quantitative data, we used the statistical packages SPSSX and LIKERT on the BYU 
Vax 8600 computer. Qualitative data (collected from interviews with students, volunteers 
and school personnel) were analyzed and categorized. Each of the following sections on 
how the project impacted students will be discussed in terms of quantitative data ai.alysis 
and qualitative data analysis. 

Achievement 

To what extent did students achieve basic math computational skills? Quantitative data 
to address this question were collected using the four criterion- referenced tests and student 
R.A.T. scores collected by the teacher. Qualitative data to address this question were 
collected through interviews. 
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Quantitative Analysis. Table 1 compares the means and standard deviations of the 
student's scores on the four criterion-referenced tests given preproject and postproject. A 
dependent T-test was administered to determine if there was a significant difference 
between the pretests and posttests results. The analysis indicated a significant change in the 
students' performance on the Fractions test, significant at the .003 level. However, the 
students' performance did not change on the other three tests. 



Table 1 

T'Tests Comparison for Pretest and Posttest Scores on Metra Tests 



Metra Test 


Mean 


Standard 
Deviation 


T Value 


Level of 
Sifnificance 


Math 1 Pretest 
Math 1 Posttest 


70.23 
70.46 


2.61 
1.81 


.16 


.872 


Math 2 Pretest 
Math 2 Posttest 


56.33 
61.08 


12.94 
8.90 


1.14 


.280 


Fractions Pretest 
Fractions Posttest 


21.50 
30.00 


8.52 
9.32 


3.72 


.003 


Decimals Pretest 
Decimals Posttest 


21.08 
19.75 


6.45 
5.88 


.75 


.467 
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Table 2 lists the students* scores for the pretest and posttest on each of the four Metra 
tests. The first number is the actual score. TTie second number in parenthesis is the percent 
correct of the total possible. The total possible points for the pretest and posttest for Math 1 
test were 74, Math 2 were 85, Fractions were 53 and Decimals were 36. Some of the test 
units and individual questions were deleted from the score possible because the students 
nor the testors did not understand the questions. 



Table 2 

Individual Student Scores (including Group Averages) and Percent Correct of Total 
P ossible Points on Metra Tests (percent correct scores are in parenthesis) 



Student 


Math 1 
Pretest 


Math 1 
Posttest 


Math 2 
Pretest 


Math 2 
Posttest 


Fractions 
Pretest 


Fractions 
Posttest 


Decimals 
Pretest 


Decimals 
Posttest 


1 


71(96) 


69(93) 


78(92) 


52(61) 


19(36) 


29(55) 


11(31) 


20(56) 


2 


71(96) 


72(97) 


59(69) 


69(81) 


38(72) 


47(89) 


27(75) 


26(72) 


3 


65(88) 


70(95) 


53(62) 


64(75) 


14(26) 


38(72) 


21(58) 


29(81) 


4 


68(92) 


72(97) 


69(81) 


65(76) 


22(42) 


30(57) 


23(64) 


23(64) 


5 


69(93) 


72(97) 


34(40) 


52(61) 


18(34) 


22(42) 


18(50) 


16(44) 


6 


72(97) 


73(99) 


66(78) 


75(88) 


27(51) 


40(75) 


36(100) 


28(78) 


7 


70(95) 


72(97) 


42(49) 


54(64) 


26(49) 


18(34) 


19(53) 


11(31) 


8 


73(99) 


68(92) 


68(80) 


68(80) 


31(58) 


33(62) 


18(50) 


16(44) 


9 


68(92) 


70(95) 


43(51) 


68(80) 


5(09) 


19(36) 


14(39) 


16(44) 


10 


74(100) 


68(92) 


60(71) 


46(54) 


16(30) 


22(42) 


19(53) 


17(47) 


n 


72(97) 


69(93) 


56(66) 


55(65) 


19(36) 


24(45) 


25(69) 


13(36) 


12 


MD* 


66(89) 


48(56) 


65(76) 


23(43) 


38(72) 


22(61) 


22(61) 


Average 
for all 
Students 


70(95) 


70(95) 


56(66) 


61(72) 


22(41) 


30(57) 


21(59) 


20(56) 



MD means missing data. 
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Table 3 compares preproject student scores on the ^standardized R.A.T. test with 
postproject student scores. The mean score is the average grade level for the pretest and 
posttest results. As the table shows, there was a significant difference between preproject 
and postproject grade level scores. Students increased a Hide less than a full grade level 
during the school year. 



Table : 

"^-T ^sts Comparison for Pretest and Posttest Scores on the R.A.T. Sta n dardized Test 



Metra Test 


Mean 


Standard 
Deviation 


T Value 


Level of 
Significance 


R.A.T. Pretest 


4.88 


.55 












2.36 


.046 


R.A.T. Posttest 

■ 


5.52 


1.24 







Qualitative Aualvsis. Most of the students (9 out of 1 1) felt that they had learned more 
about math and how to do math ir. this program vs. the previous seme.ster without the 
intervention of volunteers and computers. One student commented that she had learned 
additional math skills from the computers not taught in the class, such as measurement 
skills. However, another st'-dent indicated that he had learned more in the previous 
semester because the book^ were used mure than in this project. 

It was difficult for the volunteers to determine whether or not the students learned the 
math skills they were suppose to learn because the volunteers did not know for sure what 
was required of the students. In addition, two of the volunteers said that they did not work 
with enough students to make a judgement. One volunteer felt that the students did not 
leam what they could have learned, mainly because the of the disorganization of the 
program. Another volunteer simply indicated yes the students learned what they were 
suppose to leam, but didn*t give any rationale for his response. 

Both of the school personnel interviewed indicated that they thought students learned 
more basic matii skills as a result of this intervention than the tradition math class approach 
used the previous semester. 

Attitude about Math 

To v/hat extent did the project influence the students in terms of their attitudes about 
math? Quantitative da'ui io answer this question were cliected using the Attitude About 
School Scale. Qualitative data to address this question were collected through interviews. 
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Q uantitative Analysis. Table 4 compares the means and standard deviations of the 
students* responses to the preproject and^postprcject measures of attitude. The pretest and 
posttest means for all school subjects are very close. These results suggests that there was 
no measurable change in student attitudes as a result of the project. Apparently, the 
intervention did not have a positive or negative affect on the students' attitudes about math. 
One thing we can learn from these results is that the project did not cause negative feelings 
towarus math. These results also indicate that there was not a general attitude change 
during the study towards school in general. 



Table 4 

T-Tests Comparisons of Students* Preproject and Postproject Attitudes Toward Math, 
English, Social Studies, and Science 



Attitudinal Obiect 


Mean 


Standard 
Deviation 


T Value 


Level of 
Significance 


Math Pretest 
Math Posttest 


34.73 
34.18 


8.1 
8.6 


.37 


.720 


English Pretest 
English Posttest 


35.82 
36.27 


2.52 
8.36 


,27 


.793 


Social Stu Pretest 
Social Stu Posttest 


33.64 
32.91 


10.11 
12.t>v/ 


.33 


.747 


Science Pretest 
Science Posttest 


28.37 
26.46 


7.33 
5.61 


1.55 


.152 
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Table 5 lists the number of items, potential range of scores, and the reliability estimates 
and for each of the affective scales. The higher the score, the more positive a student's 
attitude was about math, and visa versa, a lower the score meant a student had more a 
negative attitude about math. 

Tables 

Reliability Estimates for Affective Measures 



Scale 


Number of 
of Items 


Potential Range 
of Scores 


Coefficient Alpha 
Index of 
Reliability 


Math Pretest 


14 


0-56 


.847 


Math Posttest 


14 


0-56 


.897 


English Pretest 


14 


0-56 


.792 


English Posttest 


14 


0-56 


.878 


Social Studies Pretest 


14 


0-56 


.895 


Social Studies Posttest 


14 


0-56 


.931 


Science Pretest 


14 


0-56 


.702 


Science Posttest 


14 


0-56 


.633 



14 



Table 6 lists the students* scores for tlie p -etest and posttest results on each of the 
attitude scales about school subjects. The students* scores on each of the pretest scales 
ranged from: Math 22 to 53; English 18 to 51, Social Studies 16 to 46; and Science 17 to 
44. Their scores on the posttests ranged from: Math 25 to 50; English 29 to 52; Social 
Studies 14 to 50; and Science 18 to 44. 

Table 6 

Individual Student Scores (and Group Averages) Toward Math, English. Social Studies, 
and S ci ence Affective Scales 



Math Math English English Social Stu Social Stu Science Science 
Student Pretest Posttest Potest Posttest Pretest Posttest Pretest Posttest 



1 


32 


30 


37 


33 


29 


23 


30 


26 


2 


38 


42 


36 


32 


38 


31 


27 


24 


3 


37 


41 


42 


44 


44 


50 


31 


31 


4 


37 


40 


39 


41 


38 


39 


28 


31 


5 


35 


23 


39 


45 


17 


14 


29 


26 


6 


37 


33 


18 


31 


40 


28 


19 


19 


7 


28 


25 


33 


25 


30 


28 


17 


18 


8 


53 


50 


51 


52 


46 


50 


44 


36 


9 


40 


48 


40 


50 


40 


50 


32 


44 


10 


22 


26 


39 


38 


41 


37 


24 


29 


11 


37 


38 


31 


29 


31 


46 


28 


21 


12 


26 


28 


29 


29 


16 


16 


35 


30 



Average 
for all 

Students* 34.7 34.2 35.8 36.3 33.6 32.9 28.4 26.5 
* These averages exclude student no. 9 (see Threats to Validity section). 
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Figure 1 shows how the previously listed affective scores fit on the affective 
continuum underlying the different school subjects. For example, the average score for 
students' responses on the math posttest scale was 34. This score would fall in the Neutral 
Attitude zone, however, it is very close to the Positive Attitude zone. We can conclude 
from these results that at the completion of the project, the students generally had somewhat 
of a neutral attitude towards math, with a tendency towards the positive direction. 
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Figure 1 

Continuum of Students* Attitudes Toward School Subjects 



Qualitative Analysis. All of the students except three said that their attitude towards 
nath has become more positive as result of this project. Three students said their attitude 
has stayed the same. From the student responses, the project did not have a negative affect 
on the students* attitudes towards math. One of the more positive responses was, *'More 
positive attitude. I use to hate math. Now I feel better about being in the class.*' 

Most of the volunteers indicated that they thought students attitudes about math were 
more positive as a result of the pioject. One volunteer indicated that students didn't like 
being required to go to the eighth period. Another volunteer responded by saying that he 
didn*t really know and couldn't identify any general trends among the students. 

Both of the school personnel who responded indicated that they noticed a change in 
students* attitude about math. They felt students were more positive about math as a result 
of the intervention. One commented, **Yes. Pats on the backs from volunteers helped. 
The situation was less stt-essful to the students because of the project." 

Math Anxiety 

To what extent did the project influence the students in terms of their anxiety 
towards math? Quantitative data to answer this question were collected using the Math 
Anxiety Scale. The questions in this instrument assessed the students' feelings and fears 
about math. Qualitative data to address this question were collected through interviews. 
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Q uantitative Analysis. Table 7 compares the means and standard deviations of the 
students' responses to the preproject and postproject measures of math anxiety. The pretest 
and posttest means are very close. These results suggests that there was no measurable 
change in students' anxiety towards math as a result of the project. Apparently, the 
intervention did not have a positive or negative affect on the students' anxiety towards 
math. As the case with attitude about math, these results indicate that this project did not 
increase students' anxiety towards math. 



Table 7 

T-Tests Comparisons of Students' Preproject and Postproject Anxiety Toward Math 



Attitudinal Object 


Mean 


Standard 
Deviation 


T Value 


Level of 
Significance 


Math Pretest 
Math Posttest 


26.72 
27.64 


11.63 
13.38 


.34 


.739 



Table 8 lists the number of items, potential range of scores, and the reliability 
estimates and for the pretests and posttests Math Anxiety scales. The higher the score, the 
more anxiety a student felt about math, and visa versa, the lower the score, the less anxiety 
a student felt about math. 



Table 8 

Reliability Estimates for Math Anxiety Measures 









Coefficient Alpha 




Number of 


Potential Range 


Index of 


Scale 


of Items 


of Scores 


Reliability 


Math Pretest 


18 


0-72 


.879 


Math Posttest 


18 


0-72 


.931 
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Table 9 lists the students* scores for the pretest and posttest results on each of Math 
Anxiety Scale. The students' scores on the pretest scale ranged from 1 1 to 49. Their 
scores on the posttest ranged from 7 to 60. 



Table 9 

Individual Student Scores (and Group Averages) on the Math Anxiety Scale 



Student 


Math 
Pretest 


Math 
Posttest 


1 


23 


31 


2 


18 


16 


3 


19 


18 


4 


20 


24 


5 


30 


46 


6 


18 


22 


7 


40 




8 


11 


7 


9 


17 


60 


10 


49 


49 


11 


27 


30 


12 


39 


19 


Average 
for all 
Students* 


26.7 


27.6 



* These averages exclude student no. 9 (see Threats to Validity section). 
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Like figure 1, figure 2 shows how the previously listed anxiety scores fit on the 
anxiety continuum underlying this construct. For example, the average score for students' 
responses on the math anxiety posttest scale v/as 27.6. This score would fall in the Neutral 
Attitude zone, however, it is very close to the Lx)w Anxiety zone. We can conclude from 
these results that at the completion of the project, the students generally felt neutral anxiety 
towards math, with a tendency towards the low anxiety direction. 
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Figure 2 



Continuum of Students' Anxiety Towards Math 



Qualitative Analysis. Students were not asked if they thought their anxiety towards 
math had decreased as a result of the project We thought the volunteers and school 
Personnel would be the best judges of this construct. 

Either the volunteers didn't want to make a judgement on this construct or they thought 
the students' anxiety had decreased from the beginning of the semester. Two volunteers 
felt that they could not judge. Three felt that students' anxiety had decreased. Onereponed 
that he saw very little anxiety' ^v nong students at all. One of the more positive comments 
were, "Yes. One kid in particular. I heard him talking about his lack of fear of math this 
semester." 

The two school personnel who responded to this issue gave conflicting responses. 
One responded, "Yes. Some drastic, some small amount. Some were paranoid about 
certain areas of math." The other said, "No, probably not" 

Study Habits 

To what degree did the project affect math homework study habits of the students? 
Quantitative data to answer tiiis question were collected using the Student Questionnaire. 
This questionnaire asked the students three questions concerning their study behavior 
habits. Qualitative data to address this question were collected through interviews. 
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Quantitative Analvsis^ Table 10 lists the students* responses to the first question on 
the questionnaire, which asked, "How often do you work on your math homework 
(outside of your math class)?'' The results show tiiat most of the students do homework 
about 1-2 times a week. There is not a significant change from tiie pretest and posttest 
results; however, it appears ti^.at a small shift occurred towards some students studying less 
after the project tlian before. 



Table 10 

Students' Responses on How Of:en Thev Do Homework Outside of Their Math Class 



Response Catef ories 


Pretest R 
Number of 
Respondents 


esults 

Percentage of 
Respondents 


Eqsuss 

Number of 
Respondents 


t Results 
Percentage of 
Respondents 


Everyday 


1 


8% 


0 


0% 


3-4 times a week 


1 


8% 


0 


0% 


1-2 times a week 


9 


75% 


8 


67% 


1-2 times a month 


1 


8% 


2 


17% 


Don't do homework 


0 


0% 


2 


17% 


Total 


12 


99% 


12 


101% 
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Table 1 1 list the students' responses to second question on the questionnaire, which 
asked, "On average, about how much time do you spend working on your math homework 
during the week?" These results indicate that most of the students study 1-2 hours per 
week on math homework. The project did not significantly affect the amount of time 
students spend on math homework. It appears from the data that a minor shift in behavior 
took place during the project, with a few students studying less time than at the beginning 
of the project. 



Table 11 

Students* Responses on How Much Time Thev Spend Working on Math Homework 
During a Normal Week 



Response Categories 


Pretest Results 
Number of Percentage of 
ResDOPdents Resoondents 


Posttest Results 
Number of Percentage of 
Respondents Resoondents 


None 


2 


17% 


3 


25% 


1-2 hours 


7 


58% 


8 


67% 


3-4 hours 


2 


17% 


1 


8% 


5-6 hours 


1 


8% 


0 


0% 


7-10 hours 


0 


0% 


0 


0% 


More tlian 10 hrs a week 


0 


0% 


0 


0% 


Total 


12 


100% 


12 


100% 
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Table 12 lists the students' responses to third question on the questionnaire, which 
asked, "How often do you study math during the eighth hour?" The purpose of this 
question was to determine whether students came in to study math on their own initiative as 
a result of the project. The results show that most of the students study 1-2 per week 
during the eighth hour. The project did not significantly affect the amount of time students 
study their math home\york during the eighth hour. An analysis of the pretest and posttest 
results show a slight shift towards less time being spent studying during the eighth hour at 
the end of the project. 



Table 12 

Students' Responses on How Often Thev Study Math During the Eighth Hour 



Pr etest Results Posttest Results 

Number of Percentage of Number of Percentage of 
Response Categories Respondents Respondents Respondents Respondents 



Everyday 


1 


8% 


0 


0% 


3-4 times a week 


1 


8% 


1 


8% 


1-2 times a week 


6 


50% 


5 


42% 


1-2 times a month 


3 


25% 


4 


33% 


Don't do homework 


1 


8% 


2 


17% 


Total 


12 


99% 


12 


100% 



Qualitative Analysis> Eight students indicated that their study habits improved as a 
result of the project One student commented, "Yes. I was failing before, now Tm doing 
real well. I do a page of math ev^y night." Three students said their study habits have not 
changed. Another student said, "Not really. I find more opportunity to goof off." 

One of the school personnel responded to the question of whether or not the students* 
study habits have been improved by saying, "Yes. They do their homework better than 
they did before. They are more confident in their homework because they have had a lot 
more practice because of the computer." Only one person from the school was asked a 
question about his or her perceptions on study habits. 

Positive Reinforcement 

To what extent did the project provide students with positive I'einforcement? 
Quantitative data to answe/ this question were collected using the Student Questionnaire. 
This questionnaire asked the students the following question, "How often in the past week 
has an adult at school told you that you have done well in math?" Qualitative data to 
address this question were collected through interviews. 
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Quantitative Analysis. Table 13 lists the siiidents' responses to this question. The 
results indicate that most of the students never received positive reinforcement from an 
adult (during the previous week that they responded to the questionnaire) about their 
performance in math. The pretest and posttest comparison shows that four students 
responded in the pretest thai they received positive reinforcement often or very often. 
However, none of the students responding in the posttest indicated that they had received 
often or very often positive reinforcement. 



Table 13 

Students* Responses on How Often Thev Received Positive Reinforcement from an Adult 



Pretest Results Posttest Results 

Number of Percentage of Number of Percentage of 
Response CatggOries Respondents Respondents Respondents Respondents 



Never 6 50% 7 58% 

Sometimes (1-2 times/wk) 2 17% 5 42% 

Often (about once a day 2 17% 0 0% 

Very Often (many times a day) 2 17% 0 0% 



Total 12 101% 12 100% 



Oualitative Analysis. All of the volunteers said that they tried to give positive 
reinforcement to the students. Some volunteers said they gave the students positive 
reinforcement every time time they worked with a student 

One of the school personnel responded to the question of whether or not the students' 
received more positive reinforcement in this project than the previous semester by saying, 
"Yes. Most definitely. Volunteers were great in giving the students praise. They were 
always telling me how good the students were doing.*' Only one person from the school 
was asked a question about positive reinforcement. 

Individual Help from an Adult 

To what extent did the project provide students with individual help from an adult on 
their math problems? Quantitative data to answer this question were collected using tht 
Student Questionnaire. Two questions were asked students concerning to what extent they 
received help from adults. Qualitative data to address this question were collected through 
interviews. 
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Ouanritarive Analysis. Table 14 lists the students* responses to a question which 
asked, "How often do you receive individual help from an adult at school to solve math 
problems?" Most of the students sometimes (1-2 times a week) receive help from an adult 
to solve their math problems. The pretest results indicate all of the students receive help 
from an adult at least weekly. However, the posttest results indicate that two of the 
students felt like they never had adult help in solving math problems. 



Table 14 



Students' Responses on How Often Thev Received Individual Help from an Adult to Solve 
Math Problems 



Pretest Results 


Posttest Results 


Number of 


Percentage of 


Number of 


Percentage of 


Response Categories Respondents 


Respondents 


Respondents 


Respondents 


Never 0 


0% 


2 


17% 


Sometimes (1-2 times/wk) 8 


67% 


6 


50% 


Often (about once a day) 3 


25% 


4 


33% 


Very Often (many times a day) 1 


8% 


0 


0% 


Total 12 


100% 


12 


101% 
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Table 15 reports the. results of a similar question to the previous question, however, 
with a little different emphasis. This question asked, '*How often do you have problems 
with math that you cannot solve by yourself and don't receive help from an adult at 
school?" The information we wanted from this question was the frequency when a student 
couldn*i solve a problem and obtain help from an adult. These results show that most of 
the students arc never faced with such a situation that when they ap; faced with a math 
problem they cannot solve, they cannot find help from an adult to assist them in solving the 
problem. There doesn't seem to be a significant change from the pretest and posttest 
results. However, the posttest results seem to be somewhat more positive than the pretest 
in that the pretest results had more students who had the situation occur more frequently. 



Table 15 

Students' Responses on How Often Thev Cannot Solve M at h Problems and Cannot 
Receive any Help from Aduhs 



Protest R?§lilts 


Posttest Results 


Number of 


Percentage of 


Number of 


Percentage of 


Response Categories Respondents 


Respondents 


Respondents 


Respondents 


Never 6 


50% 


6 


50% 


Sometunes (l-2times/wk) 3 


25% 


5 


42% 


Often (about once a day) 3 


25% 


1 


8% 


Very Often (many times a day) 0 


0% 


0 


0% 


Total 12 


100% 


12 


100% 



Qualit ati ve Analysis. The volunteers indicated that they did not give a lot of 
individual help to students. Three students came to the school one day a week, in which 
they helped students during that period. Two other volunteers said that they didn^t tutor 
much or not that often. The last volunteer said that he tutored about 15 percent of the time 
that he was at the school. 

One of the school personnel responded to the question of whether or not he or she 
thought the students received more individualized help in this project than the previous 
stmester using the traditional approach by saying, "Yes. Most important thing that the 
students got from the project. This was the best component of the project. The pilot group 
is near tlie normal 7th grade level math class. All the 7th graders in the class will go into 
the normal 8th grade class next year except one. One student will go into Pre-algebra." 
Only one person from the school was asked a question about individual help. 

Participantj* Perceptions of the Project 

What are the strengths and weaknesses of the project as perceived by the participants 
and what suggestions do they have concerning how the project can be improved? 
Qualitative data collected through personal interviews. Volunteer Tutor Activity Sheets, and 
field notes v/ere analyzed in this section of the report to identify participants' perceptions of 
the strength and weaknesses of the project. In addition, suggestions fror participants are 
also listed. 
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Weaknesses of the Project 

The students were not nearly as specific as the volunteers or school personnel about 
the weaknesses of the program. Six students said either "nothing" or "I don't know." 
Two students said a weakness of the program v , not having more time in the lab. One 
studen; nad a difficult time getting along with some of the volunteers and the otlier .cudent 
did not like doing math problems. 

The volunteers were very specific about some of the weaknesses and frustrations of 
the project. The following weaknesses were listed by the volunteers. 

• The disorganization of the project. 

• Didn't get to tutor students one-on-one. 

• The school's rcrating class schedule. 

• Sitting around v/aiting for something to do. 

• Difficult to supervise students in the lab with only one volunteer. 

• Volunteers were not trained or oriented on what they should be doing in the 
project. 

• Poor training and orientation. Volunteers were not trained on how to tutor 
students and run the computers. 

• Software programs didn't match the learning objectives of the class. 

• No management systc n so that they could know what programs students should 
run. 

• Vast majority of software programs didn't help much, there were more games than 
instructional. 

• The students really were not learning anything new, just playing around. The 
students were not progressing as fast as they could have. 

• The achievement tests used did not match the software or the learning objectives of 
the class. 

• The volunteers had a misconception about the entire project because an appropriate 
orientation and overview wasn't conducted. Not having a real concept of what we 
were trying to do in this project or what we were suppose to do as volunteers. 

• Poor software in the computer lab. 

• There weren't enough people involved in the proje . to run it and make it 
systematic and organized. 

• Not much support fi-om the school staff. 

• Lack of communication. 

• School should have communicated things better. 

• Very advanced students running simple math and fractions software games. 

The school personnel identified some of the same weaknesses as those listed 
previously. The following list identifies wc Jcnesses which were nOt addressed in the 
previous list by the volunteers. 

• Trying to run the computer lab wit )ut volunteers supervising the lab. 

• School Administration conducts a ! minute thing, like an assembly or 
something, and throws the tutoring and computer lab schedule completely off. 

• District support not complete. 

Strengths of the Program 

To determine what students thought were some of the strengths of the project, we 
asked them, "What did you like most about 'his program?" Seven students said the> liked 
using the computers. Two students said they really liked the variety the program offered. 
One student indicated that he liked gening out of class the most. The last student 
commented, "The volunteers taking the time for us to help us out." 
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The volunteers were asked, "What did you like most about this program?" Four of the 
six volunteers responded by saying they liked the association of the students. Another 
volunteer said he liked the to see the students having a positive experience with math. The 
last volunteer said that he liked the "theory or concept of the whole idea. It is a worthwhile 
idea to try out and develop a program." 

Two of the school personnel were asked what they liked most about the project The 
first responded, "Volunteers, they were the strength of the program. The volunteers were 
very willing. Without them we wouldn*t have had the computer lab. I wouldn*t have a 
computer lab without volunteers." The second person said, "Other teachers feel better 
about using volunteers in their classroom because of the results of this program." 

Suggestions to Improve the Project 

To determine what suggestions students had to improve the project, we asked them, 
"What suggestions do you have to improve this program?" Five students suggested 
purchasing better software for the lab. For ';xample, one student said, "Get joy sticks for 
the computers. Purchase better games, it is a drag using the keyboard wheu ycu could use 
joy sticks." Two of these five students mentioned specifically purchasing bette* games for 
the lab. One student suggested that the lab needed to be better supervised. She said, "The 
teacher should stay in the classroom, when she doesn^t, the volunteers don*t manage the 
lab properly. Students change the programs that she tells them to run as soon as she leaves 
the computer lab." Two students suggested having more volunteers to help students. 
Three students said that they couldn*t think of any^iing. The last student thought the math 
problems on the computer should be harder. 

The volunteers were asked, "What suggestions do you have to improve this program?" 
The following list is a summary of their suggestions. 

• Better organize the whole project This item was mentioned by almost every 
volunteer. 

• Have a programmer write software programs which map onto the achie'. ement 
tests and learning objectives for the class. 

• Properly instruct and train the tutors. Train them how to use the computers and 
tutor students. This item was mentioned firequently by almost all volunteers. 

• Give the volunteers a better o%^erview and focus of the concept of what the project 
is about. Need more and better orientation for volunteers. 

• Give volunteers more responsibilities. Teach them how to properly evaluate 
students progress and give them more responsibility for student learning and 
progress. 

• Map the achievement tests to the learning objectives. Then map the software onto 
the achievement tests and learning objectives. 

• Get better achievement tests. 

• Have more than one volunteer in the computer lab at one time. 

• Give the volunteers the same text that the students are using (or better yet a 
teacher s guide with the answers in it) so they can be prepared for the type of 
things the students will ask them. 

• Give the volunteers a list of the students* names so they can call them by name and 
become more personable with them. 

• Train the volunteers on the math content going to be covered that semester or year 
before school starts. 

• Establish goals with each student so they can be part of the process. 

• Hire someone full-time to run the computer lab. This person should take pride and 
full responsibility for the program and the computer lab. 

• Get better softwait. 

The school personnel were asked to list the suggestions they thought would improve 
the project. The first person suggested, "Get more and better software for the computer 



lab. Need to get diagnostic programs to determine in what areas students are weak. Then 
have student run that program where he will practice the math skills in which he is we^^k. 
Have the volunteers really control the computer lab." The second person responded, "Hire 
someone to mn the computer lab 5 hours a day. This person will take full responsibility to 
make sure the project is implemented completely and has full charge of the lab. Take a year 
to get the software mapped out according to the objectives. Ideally, the teacher would send 
the student to the computer lab with a note indicating what math skills he needs to practice, 
and the lab supervisor would then give him the appropriate software to practice those 
skills." The third person said, "More communication is needed between the volunteers and 
the teachers. Need to have an up-to-date file on each student Volunteers should sign in 
and then get right to work, no waiting time. Need much better organization." 

DISCUSSION 

This section of the report will address threats to internal validity of the study and 
discuss issues and implications concerning the major three evaluation questions. 

Limitations of the Study and Threats to Validity 

There were several constraints placed upon this evaluation which created threats to the 
internal validity of inferences about the effects of the project. In addition, because this 
evaluation was formative and included such a small sample, there are many limitations to 
which these results can be generalized. These threats and limitations and their implications 
will be discussed in the following list. 

1 . Pretest instruments were administered after the project was in progress. The Math 
Anxiety Scale, Attitude about School Scale, and the Student Questionnaire were 
administered about two weeks after the project began. Even though we requested the 
students to fill out the questionnaires as if we had given the instruments to the students 
on the first day of school (for the second semester), some students may have been 
confused and not understood this request. We should also recognize that it would 
have been impossible for tiie students to complete the questionnaires without any 
biases from the previous few weeks that the intervention had been in operation. In 
addition, the Metra achievement tests were administered by the volunteers while the 
project was in operation. A few pretests were administered as late as the beginning of 
March, about two months into the project. 

2. There were inconsistencies in hov/ the Metra pretests and posttests were administered. 
The volunteers administered the pretests over a two month period of time; however, 
the posttest were administered in one day. The volunteers only administered one 
pretest at a time, but the all four posttests were administered to students in one setting. 
The volunteers weic not as rigorous in administering the pretests as the evaluators 
were in the posttests. For example, some of the volunteers cued or prompted the 
students in solving some of the test items. Some volunteers lost track of time in timed 
items and allowed the students to continue over the maximum time of a minute. 

3 . There was not a direct match between the instructional objectives of the class (scope 
and sequence), the criterion -referenced tests and the software in the computer lab. It is 
unfair to judge a student's performance on the achievement tests if the tests did not 
measure what the teacher was teaching. For example, if a student scored 50 percent 
correct of the items on the Fractions test, we might be inclined to conclude that the 
project didn't do any good. However, if the teacher only taught 50 percent of the 
items on the test to the students, then we might say the student did very well on the 
test. The tests were designed to measure basic math computational skills. We do not 
know to what degree the teacher taught the same basic math computation skills that 
were assessed in the Metra tests. 
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4. The lack of representativeness of the students in this class posed a threat to the external 
validity of the evaluation data. The students were not randomly selected to participate, 
and this group may be very different than students in other classes. Since this group 
was so small, it is unlikely that this group is typical or representative of the students 
who will participate in this project in the future. The degree to which generalizations- 
are made from the results of this study should be severely limited. Since this was a 
formative evaluation, it was more important to identify the effects of the project on the 
students than it was to be able to generalize the results to some larger sample. 

5 . The responses for student no. 9 were not used in the data analysis of the affective data. 
Inspection of the instruments he completed showed that he responded to every item the 
same, regardless of whether it was a positive or negative Likert item. As we 
administered the f^uesticnnaires, we remembered this student completing each of the 
scales within a matter of minutes, finishing at least 10 minutes before any other 
student. 

6. The results from Volunteer Tutor Activity Sheets does not represent the activities of all 
the volunteers because only a few of tlie volunteers filled them out as they were asked. 
We asked one volunteer why she wasn*t filling out the sheet and she said that she 
didn*t want to be bothered with paper work. Two other volunteers indicated that they 
were not aware that they were suppose to be filling out the sheets everyday. Very little 
data from these sheets were used in the data analysis because they did not represent all 
the volunteers typical activities. 

7. The R.A.T. pretest was given in September, 1986, however, the project did not begin 
until January, 1987. It is possible that much of the smdent learning which resulted in 
a significant difference between the pretest and postiest results could have occurred 
previous to the implementation of the project. Also, we cannot provide information 
about how these tests were administered, and what controls were taken to ensure 
validit>' and reliability of the results. 

8 . At times it seemed as though student motivation to do their best on the tests were at a 
minimum, especially when conducting the Metra posttests. Some of the evaluators 
commented that it seemed some of the students deliberately said they could not do a 
cenain unit of the test (even though they probably could have if they had tried) so they 
could finish the test early and leave. Other students may not have taken the time to 
truthfully report how they felt about math when we conducted the affective 
postmeasures. 

9 . The positive reinforcement question on the Student Questionnaire may have reported 
incomplete data (question no. 4). The question asked students to limit their response 
to the previous week. A time frame within the last week was given because we felt 
students may not be able to remember incidences longer than that time period. Some 
problems with this is that the students may have had a bad week when we came or a 
week was too short period of time to limit their response. 

10. The trustworthiness of qualitative data (such as the data collected in interviews and field 

notes) wac not established through triangulation and member checking through 
multiple sources as described by Guba and Lincoln (1981). In addition, the 
observations should have been conducted much more frequently and for longer 
durations. These procedures were not conducted because of budget and time 
constraints. 

1 1 . The data analysis of the Syracuse Math Anxiety Scale results showed that some of the 
items nugh; have had too high of a readability level. Items which scored very well in 
previous results administered by Dr. Richard Sudweeks to upper high school students 
did not prove to be good items in these results. For example, the first item reads, "I'm 
allergic to math courses." This item scored very high using results from high school 
students; however, it did not score very well using the results from this study. 
Possibly the students could not decode the word allergic. Also, some of the items 
contained double negarives, which might have confused some of the students. 
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12. Some of the students* responses to ^he study habits questions (questions 1 through 3) 
on the Student Questionnaire conflicted. For example, a student might have responded 
that he studies math 2-3 times a week in the first question; however, then responded to 
the second question that he doesn*t spend anytime studying during the week. Students 
may have misunderstood the questions or not have read carefully enough as they were 
filling completing the questionnaire. 

After reviewing these threats to validity, you may ask, "Why do the evaluation?" 
Popham has said, "the practical constraints of many educational settings prevent us from 
designing evaluation studies as tidily as we might in a laboratory (1975, p. 200)." 
Compromises become necessary as we work with situations which involve human subjects 
and other practical considerations (Applegate, 1985). It is worth repeating here that the 
function of this evaluation was formative, to provide information aix)ut how this project 
can be improved. While it is true that generalizability from the results of this study are 
limited due to the threats of validity identified, these results provide stakeholders with the 
project improvement data expected from a formative evaluation. 

Implementation of the Project 

The project was not fully implemented as it should have been according to the original 
design. In fact, most of the components of the original design were not implemented. The 
project*s main objective to maximize the use of computers by having students execute 
specific programs commensurate with their instructional needs was not realized. Even 
though the teacher attempted to have students run specific pron-ams for their particular 
instructional ne^s, what actually happened in the computer lab did not match what should 
have happened according to the original design. 

Because there is so much variance between the original project design and what 
actually happened in the implementation of the project, making inferences about the original 
project is not appropriate. We must stress the point that the inferences and conclusions 
made in this study must be limited to the project as it was implemented and not as it was 
designed. 

Several reasons were stated earlier in this report explaining why the project was not 
implemented as it was designed. The following primary problems or issues affecting 
proper implementation are summarized. First, assignments and expectations were not 
clear. Educators were confused concerning how the project should have been implemented 
and according to what criteria. There seemed to be a lack of communication among those 
participating at the site as well as a lack of communication between those at the site and the 
district level. Related to this issue, people were not accountable for their specific 
responsibilities. No formal reporting system was established. Many of the problems 
which occurred might have been identified much earlier and overcome had such a system 
been in operation. Second, school administrators did not have the necessary time to attend 
to the project and the complete some their responsibilities. They did not realize the 
necessary time commitment this project required to successfully implement. Because other 
tasks emerged which took priority over this project, many of the tasks were uncompleted. 

Impact on Students 

What affect did the project have on student achievement? This question is very 
difficult to answer because there were so many threats to validity. What this means is that 
we need to be very cautious about any conclusions and inferences made concerning this 
project. Had data been collected with fewer threats to validity, the results of the study 
might have showed something different. But it is important to remember that the function 
of this study was formative, not summative. The quantitative data were collected and 
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statistical data analysis procedures conducted to help us understand what impact the project 
had on students and how it could be improved. 

The only criterion-referenced test which showed a statistical significance between 
pretest and posnest were tb'- results on the Fractions Test. The R.A.T. showed a 
significant difference between pretest and posttest administered at the beginning and again 
at the end of the year. There are several reasons which could explain why only the 
Fractions test was significant. For example, the teacher may have covered fractions in 
some detail in direct class instruction during most of the duration of the project. Perhaps 
students practiced fractions more frequently than other math skills because the best 
computer games deaU with fractions. Even though the R.A.T. scores showed a significant 
difference, we need to remember that the pretest were administered in September 1986, 
four months before the intervention was started. We cannot conclude from these results 
that the significance between the pretest and posttests results were caused by the 
intervention. 

The findings showed that the students seem to have a good understanding of the 
concepts and the necessary math skills for the criteria in the Math 1 test. However, their 
scores were not very high for the other three tests. The average scores for percent correct 
on the Math 2, Fractions and Decimals posttests were, respectively, 72, 57, and 56 
percent. At the conclusion of the project, most students could not perform many of the 
math computational skills required in the tests. Again, we should reiterate that the criterion- 
referenced tests did not directiy match the instructional objectives of the class. 

The qualitative data showed that most people thought the students learned more about 
math as a result of this project than the traditional approach the students had the previous 
semester. The teacher felt that the students in her pilot study class did better than her 
students in a similar basic math class primarily because of the project. 

How did the project influence student attitudes? The quantitative results showed that 
the project didn*t influence the students either positively or negatively. Before and after the 
project, students seemed to have a rather neutral attitude Gcaning more on the positive side 
than the negative) about math. 

The qualitative results suggested that almost all the participants felt that the students 
attitudes about math became more positive as a result of the project. No one mentioned that 
the project had a negative affect on students' attitudes about math. 

How did the project influence students* anxiety towards math? As with the previous 
measure, the quantitative results revealed that the project did not influence the students 
either positively or negatively. Preproject and postproject, students reported a neutral 
Geaning more on the low anxiety side than the high anxiety side) anxiety level about math. 

The volunteers and school personnel were unsure about what affect the project had on 
student*s anxiety level. Most people indicated that they couldn*t make a judgment. Some 
r 'X>ple said the project decreased some of the students' anxiety level, and a few said there 
was no change. No one indicated that the project increased student anxiety towards math. 

How did the project affect the homework study habits of the students? The 
quantitative data (obtained through self-report) suggested that the project didn't influence 
students' study habits. The interview data showed that most of tiie students said their 
study habits improved as a result of the project. The teacher also felt the students' study 
habits improved because of the project. 

To what extent did the students receive positive reinforcement? Results ftx)m the 
Student Questionnaire suggested positive reinforcement from an adult did not change much 
from preproject to postproject. Most students reported that they never received positive 
reinforcement from an adult. These results seem lo conflict with the quantitative results. 
All of the volunteers said they freely gave positive reinforcement to the students when they 
tutored them. One possible reason for this discrepancy is that many of the students were 
not tutored and did not receive praise from the volunteers. 

To what extent did the students receive individual attention? The students reported that 
they never or only sometimes received individual help. Tliese reports seem to agree with 
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the data collected from volunteer interviews. Most of the volunteers said they rarely tutored 
students. 

Strengths^ Weaknesses and Suggestions Identified bv Project Panicipants 

Despite the major weaknesses of this project, it appears that most people were 
generally pleased with it, even though it wasn'^ really the same project that was suppose to 
be implemented. Most people would like to stL* the project implemented again, but more in 
accordance to original design and using the recommendations of this study. Panicipants 
suggestions along with those of the evaJuators have been summarized and are listed in the 
next section of this repon (RECOMMENDATIONS). 

RECOMMENDATIONS 

Mostly all the participants judged this project to have a high degree of merit and worth. 
They would like to see the project continue, but with more control so it is implemented as it 
was designed. The following section lists the primary recommendations on how the 
project can be improved when it is implemented in the future. 

1 . In discussion and consultation with the participants of the project, develop a design 
document which gives a detailed description of the purpose of the project, how it is to 
function, how it is to be implemented, etc. This document should also list specific 
responsibilities and timelines when individuals should give an accounting on the status 
of their responsibilities. Someone should be responsible for tracking the 
implementation of the project to ensure that all the specific tasks are completed. 

2. Hire a computer lab supervisor. This person should take the bulk of the responsibility 
to make sure that all of the components of the project are implemented and function 
properly. He or she at a minimum should be responsible for the following 
responsibilities: (a) in consultation with teachers and other necessary people, ensure 
that the instructional objectives for the classes match the achievement tests and the 
software, (b) organize and maintain the management system, (c) supervise the activities 
of the volunteers in the lab and train them, (d) ensure that appropriate and effective 
software is purchased for the lab, (e) organize and maintain the record keeping system 
in the computer lab, and (f) work closely with the volunteer coordinator to ensure 
consistent and regular scheduling of volunteers. 

3. Train volunteers. They should not only be trained on how to effectively use the 
computers, but also how to effectively tutor srudents. Volunteers should be given 
sufficient orientation so they understand the whole picture of what the project is trying 
to accomplish and how they fit into that plan. Give the volunteers copies of the student 
and teacher materials so they can be prepared on the math skills students are learning. 
Teach volunteers the scope and sequence of the class they will be assisting. Give the 
volunteers more responsibility in the teaching and evaluating students progress. 
Volunteers do not enjoy just supervising the computer lab. They feel a great sense of 
reward and contribution by assisting students one-on-one in tutoring situations. 

4. Computer Lab. Have students work individually on the computers instead of in pairs. 
This may require reducing the time each student sits at a computer, but even this 
reduced time will be more effective than working with a partner. Furnish the computer 
lab with the necessary materials or furnishings to assist instruction. For example, 
purchase drapes to reduce the glare on the screen and prevent the sun from melting the 
disks left out. Higher chairs would help in keyboarding tasks. Keep detailed records 
of what programs students have executed and their success. This will prevent students 
ft-om only executing games the entire hour and encourage .hem to continually progress 
to more difficult problems. Purchase record keeping programs for the Apple He 
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systems in the lab to manage these records. Set up a regular schedule so that volunteers 
will at the same time everyday and not have to rotate on the school's schedule. 
5 . Eighth hour. Allow students to voluntarily come during the eighth hour to make up 
work, run favorite math games, receive individual tutor ftx)m a volunteer, etc. Most of 
the fun computer games which do not have much instructional wonh or me^lt could be 
made available for students to run during this time. 
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